Measuring the Algorithmic Convergence of Random Forests via Bootstrap Extrapolation

When making predictions with a voting rule, a basic question arises: “What is the smallest number of votes needed to make a good prediction?” In the context of ensemble classifiers, such as Random Forests or Bagging, this question represents a tradeoff between computational cost and statistical performance. Namely, by paying a larger computational price for more classifiers, the prediction error of the ensemble tends to improve and become more stable. Conversely, by using fewer classifiers and tolerating some variability in accuracy, it is possible to speed up the tasks of training the ensemble and making new predictions. In this paper, we propose a bootstrap method to quantify this tradeoff for the methods of Bagging and Random Forests. To be specific, suppose the training dataset is fixed, and let the random variable Errt denote the prediction error of a randomly generated ensemble of t = 1, 2, . . . classifiers. (The randomness of Errt comes only from the algorithmic randomness of the ensemble.) Working under a “first order model” of Random Forests, we prove that the centered law of Errt can be consistently estimated via our proposed method as t→∞. As a consequence, this result offers practitioners a guideline for choosing the smallest number of base classifiers needed to ensure that the algorithmic fluctuations are negligible, e.g. var(Errt) less than a given threshold.